Rate - Distortion - Optimized Video Coding Using a 3 - D

| Model-based video synthesis is combined with block-based motion-compensated video coding for rate-distortion eecient transmission of head-and-shoulder sequences. Two frames are utilized for block-based motion-compensated prediction of the current video frame where one frame is the previously reconstructed one and the other frame is provided by a model-based coder. The model-based coder uses a parameterized 3-D head model for the estimation of the motion as well as the facial expressions of a person. A synthetic output frame is rendered serving as a second reference frame for block-based motion compensation of a video coder. The coder is based on the H.263 syntax which is extended to multi-frame motion-compensated prediction incorporating a picture reference parameter. Rate-distortion optimization is employed for the coding control of the video coder providing robustness against model failures. Hence, the coding eeciency of the proposed combined codec never degrades below what is achievable with H.263 syntax even if the model-based coder cannot describe the current scene and the second reference frame is useless. On the other hand, if the objects in the scene are well described by the model-based coder, signiicant coding gains can be achieved. Experiments are conducted for two CIF sequences with accurate 3-D head shape information from a 3-D laser scanner. In addition to that, results for the well known test sequence Akiyo are presented where instead of an accurate 3-D scan a generic head model is used enabling low-cost applications such as video-telephony or video-conferencing. Bit-rate savings of about 35 % are achieved at equal average PSNR when comparing the proposed codec to TMN-10, the state-of-the-art test model of the H.263 standard. When encoding at equal bit-rate for both codecs, signiicant improvements in terms of subjective quality are visible.